What are Log Insights in CloudWatch?

Hi, this is Charu from Classmethod.

In this hands-on guide, we'll dive deep into CloudWatch Logs Insights, exploring its features and demonstrating how to harness its full potential. CloudWatch Logs Insights helps you to analyze and visualize logs effortlessly.

AWS CloudWatch Logs Insights is a fully managed service that allows you to interactively search and analyze your log data in real-time. It offers a powerful query language, simplified log data exploration, and customizable visualizations, enabling you to troubleshoot issues, detect anomalies, and gain operational insights quickly. Just make sure that access logging is enabled for your AWS resources like Amazon S3 (for static websites) or Elastic Load Balancing.

We will also look in detail, how to fetch an average daily lambda access count.

Let's get started!

Step 1: Accessing CloudWatch Logs Insights

Go to the AWS Management Console and navigate to the CloudWatch service.

In the left sidebar, under Logs click on Log insights.

Select the Log group containing the logs you want to analyze.

Write your query to analyze.

Step 2: Writing Queries

To retrieve all log entries:

*

To filter log entries by a specific keyword:

filter @message like /error/

A basic query to ensure you can retrieve any data at all, such as:

fields @timestamp, @message
| limit 20

To find an average daily estimated access count:

fields @timestamp, @message
| stats count(*) as DailyAccessCount by bin(24h)
| stats avg(DailyAccessCount)

This query does the following:

fields @timestamp, @message: Selects the timestamp and message fields from your logs.

stats count(*) as dailyAccessCount: Counts the number of log entries.

by bin(24h): Groups the counts into 24-hour bins (daily).

Customize your query:

Depending on the structure of your log data and what specific information you need, you might need to adjust the query. For example, if your logs contain specific fields for user IDs or session IDs, you might want to count distinct IDs:

fields @timestamp, userId
| stats count_distinct(userId) as uniqueUsers by bin(24h)

Step 3: Query Generator:

You can even generate queries automatically. You just have to write down what you want to achieve under Query generator Prompt box and click on Generate new query button. It will generate a new query automatically.

Conclusion

You've learned the basics of AWS CloudWatch Logs Insights and how to use it to analyze and visualize log data effectively. By mastering CloudWatch Logs Insights, you can gain valuable insights into your applications, troubleshoot issues efficiently, and optimize your infrastructure's performance.

Considering Lambda Logs Insights to estimate website access counts, it is feasible if your Lambda functions are directly involved in handling web requests and are properly logging these requests. Ensure logs are detailed and consistent for accurate analysis. For many setups, combining data from other AWS services (like API Gateway or ELB) might provide more comprehensive insights.

Thank you for reading!

Happy Learning:)